Facilitating execution of conceptual queries containing qualitative search terms

ABSTRACT

The disclosed embodiments relate to a system that facilitates performing searches based on qualitative search terms. During operation, the system receives a query that applies a qualitative search term to an attribute of data items in a set of data items. While executing the query, the system processes each data item in the set of data items by extracting an attribute value from the data item and then using a concept-mapping to determine a compatibility index for the attribute value, wherein the concept-mapping associates each attribute value with a numerical compatibility index that indicates a compatibility between the attribute value and the qualitative search term. Finally, the system uses the compatibility index as a factor in determining whether to include the data item in a set of query results.

RELATED APPLICATION

The subject matter of this application is related to the subject matterin a co-pending non-provisional application by the same inventors as theinstant application and filed on the same day as the instant applicationentitled “Technique for Updating a Context that Facilitates EvaluatingQualitative Search Terms,” having serial number Ser. No. 14/448,215, andfiling date 31 Jul. 2014.

BACKGROUND

1. Field of the Invention

The disclosed embodiments generally relate to techniques for searchingthrough data. More specifically, the disclosed embodiments relate to thedesign of a system that facilitates executing conceptual queries thatsearch through data using qualitative search terms, such as “short” and“tall.”

2. Related Art

Humans typically think about physical parameters using qualitativeconcepts, such as “fast” or “slow,” instead of a precise numericalvalue, such as “25.6 miles per hour.” As a consequence, when analystssearch through data, they are typically interested in identifying dataitems that satisfy qualitative criteria. For example, an analyst maywant to identify servers having an “unusually long response time.” Inexisting query-processing systems, an analyst typically translatesqualitative criteria into corresponding numerical values beforeperforming a search. This process is both time-consuming and problematicbecause the numerical values for specific qualitative search criteriacan vary considerably among different contexts. For example, thedefinition of an “unusually long response time” for a computer server islikely to be different during a peak-usage time in the middle of the dayin comparison to a low-usage time in the middle of the night. Also, thedefinition of a specific qualitative search criterion can vary amongdifferent users and different contexts.

Hence, what is needed is a system that facilitates performing searchesbased on qualitative search criteria that can vary among different usagecontexts.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 presents a block diagram of an event-processing system inaccordance with the disclosed embodiments.

FIG. 2 presents a flow chart illustrating how indexers process, index,and store data received from forwarders in accordance with the disclosedembodiments.

FIG. 3 presents a flow chart illustrating how a search head and indexersperform a search query in accordance with the disclosed embodiments.

FIG. 4 presents a block diagram of a system for processing searchrequests that uses extraction rules for field values in accordance withthe disclosed embodiments.

FIG. 5 illustrates an exemplary search query received from a client andexecuted by search peers in accordance with the disclosed embodiments.

FIG. 6A illustrates a search screen in accordance with the disclosedembodiments.

FIG. 6B illustrates a data summary dialog that enables a user to selectvarious data sources in accordance with the disclosed embodiments.

FIG. 7A illustrates an exemplary concept-mapping in accordance with thedisclosed embodiments.

FIG. 7B illustrates a set of context-mappings that comprise a context inaccordance with the disclosed embodiments.

FIG. 7C illustrates how hedges can be applied to a concept-mapping inaccordance with the disclosed embodiments.

FIG. 8 illustrates various concepts, and contexts and an associatedcontainer in accordance with the disclosed embodiments.

FIG. 9A illustrates various conceptual-search commands in accordancewith the disclosed embodiments.

FIG. 9B illustrates statistical-reasoning commands and other commands inaccordance with the disclosed embodiments.

FIG. 10 presents a flow chart illustrating how a query containing aqualitative search term can be processed in accordance with thedisclosed embodiments.

FIG. 11 presents a flow chart illustrating how linguistic hedges can beprocessed in accordance with the disclosed embodiments.

FIG. 12 presents a flow chart illustrating how compatibility indices fordifferent qualitative search terms can be combined during queryprocessing in accordance with the disclosed embodiments.

FIG. 13 presents a flow chart illustrating how a context can be updatedin accordance with the disclosed embodiments.

DETAILED DESCRIPTION

The disclosed embodiments relate to a query-processing system thatperforms searches based on qualitative search terms. Thisquery-processing system is described in more detail below, but first wedescribe the structure of an event-based system in which thisquery-processing system operates.

1.1 System Overview

Modern data centers often comprise thousands of host computer systemsthat operate collectively to service requests from even larger numbersof remote clients. During operation, these data centers generatesignificant volumes of performance data and diagnostic information thatcan be analyzed to quickly diagnose performance problems. In order toreduce the size of this performance data, the data is typicallypre-processed prior to being stored based on anticipated data-analysisneeds. For example, pre-specified data items can be extracted from theperformance data and stored in a database to facilitate efficientretrieval and analysis at search time. However, the rest of theperformance data is not saved and is essentially discarded duringpre-processing. As storage capacity becomes progressively cheaper andmore plentiful, there are fewer incentives to discard this performancedata and many reasons to keep it.

This plentiful storage capacity is presently making it feasible to storemassive quantities of minimally processed performance data at “ingestiontime” for later retrieval and analysis at “search time.” Note thatperforming the analysis operations at search time provides greaterflexibility because it enables an analyst to search all of theperformance data, instead of searching pre-specified data items thatwere stored at ingestion time. This enables the analyst to investigatedifferent aspects of the performance data instead of being confined tothe pre-specified set of data items that were selected at ingestiontime.

However, analyzing massive quantities of heterogeneous performance dataat search time can be a challenging task. A data center may generateheterogeneous performance data from thousands of different components,which can collectively generate tremendous volumes of performance datathat can be time-consuming to analyze. For example, this performancedata can include data from system logs, network packet data, sensordata, and data generated by various applications. Also, the unstructurednature of much of this performance data can pose additional challengesbecause of the difficulty of applying semantic meaning to unstructureddata, and the difficulty of indexing and querying unstructured datausing traditional database systems.

These challenges can be addressed by using an event-based system, suchas the SPLUNK® ENTERPRISE system produced by Splunk Inc. of SanFrancisco, Calif., to store and process performance data. The SPLUNK®ENTERPRISE system is the leading platform for providing real-timeoperational intelligence that enables organizations to collect, index,and harness machine-generated data from various websites, applications,servers, networks, and mobile devices that power their businesses. TheSPLUNK® ENTERPRISE system is particularly useful for analyzingunstructured performance data, which is commonly found in system logfiles. Although many of the techniques described herein are explainedwith reference to the SPLUNK® ENTERPRISE system, the techniques are alsoapplicable to other types of data server systems.

In the SPLUNK® ENTERPRISE system, performance data is stored as“events,” wherein each event comprises a collection of performance dataand/or diagnostic information that is generated by a computer system andis correlated with a specific point in time. Events can be derived from“time series data,” wherein time series data comprises a sequence ofdata points (e.g., performance measurements from a computer system) thatare associated with successive points in time and are typically spacedat uniform time intervals. Events can also be derived from “structured”or “unstructured” data. Structured data has a predefined format, whereinspecific data items with specific data formats reside at predefinedlocations in the data. For example, structured data can include dataitems stored in fields in a database table. In contrast, unstructureddata does not have a predefined format. This means that unstructureddata can comprise various data items having different data types thatcan reside at different locations. For example, when the data source isan operating system log, an event can include one or more lines from theoperating system log containing raw data that includes different typesof performance and diagnostic information associated with a specificpoint in time. Examples of data sources from which an event may bederived include, but are not limited to: web servers; applicationservers; databases; firewalls; routers; operating systems; and softwareapplications that execute on computer systems, mobile devices, andsensors. The data generated by such data sources can be produced invarious forms including, for example and without limitation, server logfiles, activity log files, configuration files, messages, network packetdata, performance measurements and sensor measurements. An eventtypically includes a timestamp that may be derived from the raw data inthe event, or may be determined through interpolation between temporallyproximate events having known timestamps.

The SPLUNK® ENTERPRISE system also facilitates using a flexible schemato specify how to extract information from the event data, wherein theflexible schema may be developed and redefined as needed. Note that aflexible schema may be applied to event data “on the fly,” when it isneeded (e.g., at search time), rather than at ingestion time of the dataas in traditional database systems. Because the schema is not applied toevent data until it is needed (e.g., at search time), it is referred toas a “late-binding schema.”

During operation, the SPLUNK® ENTERPRISE system starts with raw data,which can include unstructured data, machine data, performancemeasurements or other time-series data, such as data obtained fromweblogs, syslogs, or sensor readings. It divides this raw data into“portions,” and optionally transforms the data to produce timestampedevents. The system stores the timestamped events in a data store, andenables a user to run queries against the data store to retrieve eventsthat meet specified criteria, such as containing certain keywords orhaving specific values in defined fields. Note that the term “field”refers to a location in the event data containing a value for a specificdata item.

As noted above, the SPLUNK® ENTERPRISE system facilitates using alate-binding schema while performing queries on events. A late-bindingschema specifies “extraction rules” that are applied to data in theevents to extract values for specific fields. More specifically, theextraction rules for a field can include one or more instructions thatspecify how to extract a value for the field from the event data. Anextraction rule can generally include any type of instruction forextracting values from data in events. In some cases, an extraction rulecomprises a regular expression, in which case the rule is referred to asa “regex rule.”

In contrast to a conventional schema for a database system, alate-binding schema is not defined at data ingestion time. Instead, thelate-binding schema can be developed on an ongoing basis until the timea query is actually executed. This means that extraction rules for thefields in a query may be provided in the query itself, or may be locatedduring execution of the query. Hence, as an analyst learns more aboutthe data in the events, the analyst can continue to refine thelate-binding schema by adding new fields, deleting fields, or changingthe field extraction rules until the next time the schema is used by aquery. Because the SPLUNK® ENTERPRISE system maintains the underlyingraw data and provides a late-binding schema for searching the raw data,it enables an analyst to investigate questions that arise as the analystlearns more about the events.

In the SPLUNK® ENTERPRISE system, a field extractor may be configured toautomatically generate extraction rules for certain fields in the eventswhen the events are being created, indexed, or stored, or possibly at alater time. Alternatively, a user may manually define extraction rulesfor fields using a variety of techniques.

Also, a number of “default fields” that specify metadata about theevents rather than data in the events themselves can be createdautomatically. For example, such default fields can specify: a timestampfor the event data; a host from which the event data originated; asource of the event data; and a source type for the event data. Thesedefault fields may be determined automatically when the events arecreated, indexed or stored.

In some embodiments, a common field name may be used to reference two ormore fields containing equivalent data items, even though the fields maybe associated with different types of events that possibly havedifferent data formats and different extraction rules. By enabling acommon field name to be used to identify equivalent fields fromdifferent types of events generated by different data sources, thesystem facilitates use of a “common information model” (CIM) across thedifferent data sources.

1.2 Data Server System

FIG. 1 presents a block diagram of an exemplary event-processing system100, similar to the SPLUNK® ENTERPRISE system. System 100 includes oneor more forwarders 101 that collect data obtained from a variety ofdifferent data sources 105, and one or more indexers 102 that store,process, and/or perform operations on this data, wherein each indexeroperates on data contained in a specific data store 103. Theseforwarders and indexers can comprise separate computer systems in a datacenter, or may alternatively comprise separate processes executing onvarious computer systems in a data center.

During operation, the forwarders 101 identify which indexers 102 willreceive the collected data and then forward the data to the identifiedindexers. Forwarders 101 can also perform operations to strip outextraneous data and detect timestamps in the data. The forwarders nextdetermine which indexers 102 will receive each data item and thenforward the data items to the determined indexers 102.

Note that distributing data across different indexers facilitatesparallel processing. This parallel processing can take place at dataingestion time, because multiple indexers can process the incoming datain parallel. The parallel processing can also take place at search time,because multiple indexers can search through the data in parallel.

System 100 and the processes described below with respect to FIGS. 1-5are further described in “Exploring Splunk Search Processing Language(SPL) Primer and Cookbook” by David Carasso, CITO Research, 2012, and in“Optimizing Data Analysis With a Semi-Structured Time Series Database”by Ledion Bitincka, Archana Ganapathi, Stephen Sorkin, and Steve Zhang,SLAML, 2010, each of which is hereby incorporated herein by reference inits entirety for all purposes.

1.3 Data Ingestion

FIG. 2 presents a flow chart illustrating how an indexer processes,indexes, and stores data received from forwarders in accordance with thedisclosed embodiments. At block 201, the indexer receives the data fromthe forwarder. Next, at block 202, the indexer apportions the data intoevents. Note that the data can include lines of text that are separatedby carriage returns or line breaks and an event may include one or moreof these lines. During the apportioning process, the indexer can useheuristic rules to automatically determine the boundaries of the events,which for example coincide with line boundaries. These heuristic rulesmay be determined based on the source of the data, wherein the indexercan be explicitly informed about the source of the data or can infer thesource of the data by examining the data. These heuristic rules caninclude regular expression-based rules or delimiter-based rules fordetermining event boundaries, wherein the event boundaries may beindicated by predefined characters or character strings. Thesepredefined characters may include punctuation marks or other specialcharacters including, for example, carriage returns, tabs, spaces orline breaks. In some cases, a user can fine-tune or configure the rulesthat the indexers use to determine event boundaries in order to adaptthe rules to the user's specific requirements.

Next, the indexer determines a timestamp for each event at block 203. Asmentioned above, these timestamps can be determined by extracting thetime directly from data in the event, or by interpolating the time basedon timestamps from temporally proximate events. In some cases, atimestamp can be determined based on the time the data was received orgenerated. The indexer subsequently associates the determined timestampwith each event at block 204, for example by storing the timestamp asmetadata for each event.

Then, the system can apply transformations to data to be included inevents at block 205. For log data, such transformations can includeremoving a portion of an event (e.g., a portion used to define eventboundaries, extraneous text, characters, etc.) or removing redundantportions of an event. Note that a user can specify portions to beremoved using a regular expression or any other possible technique.

Next, a keyword index can optionally be generated to facilitate fastkeyword searching for events. To build a keyword index, the indexerfirst identifies a set of keywords in block 206. Then, at block 207 theindexer includes the identified keywords in an index, which associateseach stored keyword with references to events containing that keyword(or to locations within events where that keyword is located). When anindexer subsequently receives a keyword-based query, the indexer canaccess the keyword index to quickly identify events containing thekeyword.

In some embodiments, the keyword index may include entries forname-value pairs found in events, wherein a name-value pair can includea pair of keywords connected by a symbol, such as an equals sign orcolon. In this way, events containing these name-value pairs can bequickly located. In some embodiments, fields can automatically begenerated for some or all of the name-value pairs at the time ofindexing. For example, if the string “dest=10.0.1.2” is found in anevent, a field named “dest” may be created for the event, and assigned avalue of “10.0.1.2.”

Finally, the indexer stores the events in a data store at block 208,wherein a timestamp can be stored with each event to facilitatesearching for events based on a time range. In some cases, the storedevents are organized into a plurality of buckets, wherein each bucketstores events associated with a specific time range. This not onlyimproves time-based searches, but it also allows events with recenttimestamps that may have a higher likelihood of being accessed to bestored in faster memory to facilitate faster retrieval. For example, abucket containing the most recent events can be stored as flash memoryinstead of on hard disk.

Each indexer 102 is responsible for storing and searching a subset ofthe events contained in a corresponding data store 103. By distributingevents among the indexers and data stores, the indexers can analyzeevents for a query in parallel, for example using map-reduce techniques,wherein each indexer returns partial responses for a subset of events toa search head that combines the results to produce an answer for thequery. By storing events in buckets for specific time ranges, an indexermay further optimize searching by looking only in buckets for timeranges that are relevant to a query.

Moreover, events and buckets can also be replicated across differentindexers and data stores to facilitate high availability and disasterrecovery as is described in U.S. patent application Ser. No. 14/266,812filed on 30 Apr. 2014, and in U.S. application patent Ser. No.14/266,817 also filed on 30 Apr. 2014.

1.4 Query Processing

FIG. 3 presents a flow chart illustrating how a search head and indexersperform a search query in accordance with the disclosed embodiments. Atthe start of this process, a search head receives a search query from aclient at block 301. Next, at block 302, the search head analyzes thesearch query to determine what portions can be delegated to indexers andwhat portions need to be executed locally by the search head. At block303, the search head distributes the determined portions of the query tothe indexers. Note that commands that operate on single events can betrivially delegated to the indexers, while commands that involve eventsfrom multiple indexers are harder to delegate.

Then, at block 304, the indexers to which the query was distributedsearch their data stores for events that are responsive to the query. Todetermine which events are responsive to the query, the indexer searchesfor events that match the criteria specified in the query. This criteriacan include matching keywords or specific values for certain fields. Ina query that uses a late-binding schema, the searching operations inblock 304 may involve using the late-binding scheme to extract valuesfor specified fields from events at the time the query is processed.Next, the indexers can either send the relevant events back to thesearch head, or use the events to calculate a partial result, and sendthe partial result back to the search head.

Finally, at block 305, the search head combines the partial resultsand/or events received from the indexers to produce a final result forthe query. This final result can comprise different types of datadepending upon what the query is asking for. For example, the finalresults can include a listing of matching events returned by the query,or some type of visualization of data from the returned events. Inanother example, the final result can include one or more calculatedvalues derived from the matching events.

Moreover, the results generated by system 100 can be returned to aclient using different techniques. For example, one technique streamsresults back to a client in real-time as they are identified. Anothertechnique waits to report results to the client until a complete set ofresults is ready to return to the client. Yet another technique streamsinterim results back to the client in real-time until a complete set ofresults is ready, and then returns the complete set of results to theclient. In another technique, certain results are stored as “searchjobs,” and the client may subsequently retrieve the results byreferencing the search jobs.

The search head can also perform various operations to make the searchmore efficient. For example, before the search head starts executing aquery, the search head can determine a time range for the query and aset of common keywords that all matching events must include. Next, thesearch head can use these parameters to query the indexers to obtain asuperset of the eventual results. Then, during a filtering stage, thesearch head can perform field-extraction operations on the superset toproduce a reduced set of search results.

1.5 Field Extraction

FIG. 4 presents a block diagram illustrating how fields can be extractedduring query processing in accordance with the disclosed embodiments. Atthe start of this process, a search query 402 is received at a queryprocessor 404. Query processor 404 includes various mechanisms forprocessing a query, wherein these mechanisms can reside in a search head104 and/or an indexer 102. Note that the exemplary search query 402illustrated in FIG. 4 is expressed in the Search Processing Language(SPL), which is used in conjunction with the SPLUNK® ENTERPRISE system.SPL is a pipelined search language in which a set of inputs is operatedon by a first command in a command line, and then a subsequent commandfollowing the pipe symbol “I” operates on the results produced by thefirst command, and so on for additional commands. Search query 402 canalso be expressed in other query languages, such as the Structured QueryLanguage (“SQL”) or any suitable query language.

Upon receiving search query 402, query processor 404 sees that searchquery 402 includes two fields “IP” and “target.” Query processor 404also determines that the values for the “IP” and “target” fields havenot already been extracted from events in data store 414, andconsequently determines that query processor 404 needs to use extractionrules to extract values for the fields. Hence, query processor 404performs a lookup for the extraction rules in a rule base 406, whereinrule base 406 maps field names to corresponding extraction rules andobtains extraction rules 408-409, wherein extraction rule 408 specifieshow to extract a value for the “IP” field from an event, and extractionrule 409 specifies how to extract a value for the “target” field from anevent. As is illustrated in FIG. 4, extraction rules 408-409 cancomprise regular expressions that specify how to extract values for therelevant fields. Such regular-expression-based extraction rules are alsoreferred to as “regex rules.” In addition to specifying how to extractfield values, the extraction rules may also include instructions forderiving a field value by performing a function on a character string orvalue retrieved by the extraction rule. For example, a transformationrule may truncate a character string, or convert the character stringinto a different data format. In some cases, the query itself canspecify one or more extraction rules.

Next, query processor 404 sends extraction rules 408-409 to a fieldextractor 412, which applies extraction rules 408-409 to events 416-418in a data store 414. Note that data store 414 can include one or moredata stores, and extraction rules 408-409 can be applied to largenumbers of events in data store 414, and are not meant to be limited tothe three events 416-418 illustrated in FIG. 4. Moreover, the queryprocessor 404 can instruct field extractor 412 to apply the extractionrules to all the events in a data store 414, or to a subset of theevents that have been filtered based on some criteria.

Next, field extractor 412 applies extraction rule 408 for the firstcommand “Search IP=“10*” to events in data store 414 including events416-418. Extraction rule 408 is used to extract values for the IPaddress field from events in data store 414 by looking for a pattern ofone or more digits, followed by a period, followed again by one or moredigits, followed by another period, followed again by one or moredigits, followed by another period, and followed again by one or moredigits. Next, field extractor 412 returns field values 420 to queryprocessor 404, which uses the criterion IP=“10*” to look for IPaddresses that start with “10”. Note that events 416 and 417 match thiscriterion, but event 418 does not, so the result set for the firstcommand is events 416-417.

Query processor 404 then sends events 416-417 to the next command “statscount target.” To process this command, query processor 404 causes fieldextractor 412 to apply extraction rule 409 to events 416-417. Extractionrule 409 is used to extract values for the target field for events416-417 by skipping the first four commas in events 416-417, and thenextracting all of the following characters until a comma or period isreached. Next, field extractor 412 returns field values 421 to queryprocessor 404, which executes the command “stats count target” to countthe number of unique values contained in the target fields, which inthis example produces the value “2” that is returned as a final result422 for the query.

Note that query results can be returned to a client, a search head, orany other system component for further processing. In general, queryresults may include: a set of one or more events; a set of one or morevalues obtained from the events; a subset of the values; statisticscalculated based on the values; a report containing the values; or avisualization, such as a graph or chart, generated from the values.

1.6 Exemplary Search Screen

FIG. 6A illustrates an exemplary search screen 600 in accordance withthe disclosed embodiments. Search screen 600 includes a search bar 602that accepts user input in the form of a search string. It also includesa time range picker 612 that enables the user to specify a time rangefor the search. For “historical searches” the user can select a specifictime range, or alternatively a relative time range, such as “today,”“yesterday” or “last week.” For “real-time searches,” the user canselect the size of a preceding time window to search for real-timeevents. Search screen 600 also initially displays a “data summary”dialog as is illustrated in FIG. 6B that enables the user to selectdifferent sources for the event data, for example by selecting specifichosts and log files.

After the search is executed, the search screen 600 can display theresults through search results tabs 604, wherein search results tabs 604includes: an “events tab” that displays various information about eventsreturned by the search; a “statistics tab” that displays statisticsabout the search results; and a “visualization tab” that displaysvarious visualizations of the search results. The events tab illustratedin FIG. 6A displays a timeline graph 605 that graphically illustratesthe number of events that occurred in one-hour intervals over theselected time range. It also displays an events list 608 that enables auser to view the raw data in each of the returned events. Itadditionally displays a fields sidebar 606 that includes statisticsabout occurrences of specific fields in the returned events, including“selected fields” that are pre-selected by the user, and “interestingfields” that are automatically selected by the system based onpre-specified criteria.

1.7 Acceleration Techniques

The above-described system provides significant flexibility by enablinga user to analyze massive quantities of minimally processed performancedata “on the fly” at search time instead of storing pre-specifiedportions of the performance data in a database at ingestion time. Thisflexibility enables a user to see correlations in the performance dataand perform subsequent queries to examine interesting aspects of theperformance data that may not have been apparent at ingestion time.

However, performing extraction and analysis operations at search timecan involve a large amount of data and require a large number ofcomputational operations, which can cause considerable delays whileprocessing the queries. Fortunately, a number of acceleration techniqueshave been developed to speed up analysis operations performed at searchtime. These techniques include: (1) performing search operations inparallel by formulating a search as a map-reduce computation; (2) usinga keyword index; (3) using a high performance analytics store; and (4)accelerating the process of generating reports. These techniques aredescribed in more detail below.

1.7.1 Map-Reduce Technique

To facilitate faster query processing, a query can be structured as amap-reduce computation, wherein the “map” operations are delegated tothe indexers, while the corresponding “reduce” operations are performedlocally at the search head. For example, FIG. 5 illustrates how a searchquery 501 received from a client at search head 104 can split into twophases, including: (1) a “map phase” comprising subtasks 502 (e.g., dataretrieval or simple filtering) that may be performed in parallel and are“mapped” to indexers 102 for execution, and (2) a “reduce phase”comprising a merging operation 503 to be executed by the search headwhen the results are ultimately collected from the indexers.

During operation, upon receiving search query 501, search head 104modifies search query 501 by substituting “stats” with “prestats” toproduce search query 502, and then distributes search query 502 to oneor more distributed indexers, which are also referred to as “searchpeers.” Note that search queries may generally specify search criteriaor operations to be performed on events that meet the search criteria.Search queries may also specify field names, as well as search criteriafor the values in the fields or operations to be performed on the valuesin the fields. Moreover, the search head may distribute the full searchquery to the search peers as is illustrated in FIG. 3, or mayalternatively distribute a modified version (e.g., a more restrictedversion) of the search query to the search peers. In this example, theindexers are responsible for producing the results and sending them tothe search head. After the indexers return the results to the searchhead, the search head performs the merging operations 503 on theresults. Note that by executing the computation in this way, the systemeffectively distributes the computational operations while minimizingdata transfers.

1.7.2 Keyword Index

As described above with reference to the flow charts in FIGS. 2 and 3,event-processing system 100 can construct and maintain one or morekeyword indices to facilitate rapidly identifying events containingspecific keywords. This can greatly speed up the processing of queriesinvolving specific keywords. As mentioned above, to build a keywordindex, an indexer first identifies a set of keywords. Then, the indexerincludes the identified keywords in an index, which associates eachstored keyword with references to events containing that keyword, or tolocations within events where that keyword is located. When an indexersubsequently receives a keyword-based query, the indexer can access thekeyword index to quickly identify events containing the keyword.

1.7.3 High Performance Analytics Store

To speed up certain types of queries, some embodiments of system 100make use of a high performance analytics store, which is referred to asa “summarization table,” that contains entries for specific field-valuepairs. Each of these entries keeps track of instances of a specificvalue in a specific field in the event data and includes references toevents containing the specific value in the specific field. For example,an exemplary entry in a summarization table can keep track ofoccurrences of the value “94107” in a “ZIP code” field of a set ofevents, wherein the entry includes references to all of the events thatcontain the value “94107” in the ZIP code field. This enables the systemto quickly process queries that seek to determine how many events have aparticular value for a particular field, because the system can examinethe entry in the summarization table to count instances of the specificvalue in the field without having to go through the individual events ordo extractions at search time. Also, if the system needs to process allevents that have a specific field-value combination, the system can usethe references in the summarization table entry to directly access theevents to extract further information without having to search all ofthe events to find the specific field-value combination at search time.

In some embodiments, the system maintains a separate summarization tablefor each of the above-described time-specific buckets that stores eventsfor a specific time range, wherein a bucket-specific summarization tableincludes entries for specific field-value combinations that occur inevents in the specific bucket. Alternatively, the system can maintain aseparate summarization table for each indexer, wherein theindexer-specific summarization table only includes entries for theevents in a data store that is managed by the specific indexer.

The summarization table can be populated by running a “collection query”that scans a set of events to find instances of a specific field-valuecombination, or alternatively instances of all field-value combinationsfor a specific field. A collection query can be initiated by a user, orcan be scheduled to occur automatically at specific time intervals. Acollection query can also be automatically launched in response to aquery that asks for a specific field-value combination.

In some cases, the summarization tables may not cover all of the eventsthat are relevant to a query. In this case, the system can use thesummarization tables to obtain partial results for the events that arecovered by summarization tables, but may also have to search throughother events that are not covered by the summarization tables to produceadditional results. These additional results can then be combined withthe partial results to produce a final set of results for the query.This summarization table and associated techniques are described in moredetail in U.S. Pat. No. 8,682,925, issued on Mar. 25, 2014.

1.7.4 Accelerating Report Generation

In some embodiments, a data server system such as the SPLUNK® ENTERPRISEsystem can accelerate the process of periodically generating updatedreports based on query results. To accelerate this process, asummarization engine automatically examines the query to determinewhether generation of updated reports can be accelerated by creatingintermediate summaries. (This is possible if results from preceding timeperiods can be computed separately and combined to generate an updatedreport. In some cases, it is not possible to combine such incrementalresults, for example where a value in the report depends onrelationships between events from different time periods.) If reportscan be accelerated, the summarization engine periodically generates asummary covering data obtained during a latest non-overlapping timeperiod. For example, where the query seeks events meeting a specifiedcriteria, a summary for the time period includes only events within thetime period that meet the specified criteria. Similarly, if the queryseeks statistics calculated from the events, such as the number ofevents that match the specified criteria, then the summary for the timeperiod includes the number of events in the period that match thespecified criteria.

In parallel with the creation of the summaries, the summarization engineschedules the periodic updating of the report associated with the query.During each scheduled report update, the query engine determines whetherintermediate summaries have been generated covering portions of the timeperiod covered by the report update. If so, then the report is generatedbased on the information contained in the summaries. Also, if additionalevent data has been received and has not yet been summarized, and isrequired to generate the complete report, the query can be run on thisadditional event data. Then, the results returned by this query on theadditional event data, along with the partial results obtained from theintermediate summaries, can be combined to generate the updated report.This process is repeated each time the report is updated. Alternatively,if the system stores events in buckets covering specific time ranges,then the summaries can be generated on a bucket-by-bucket basis. Notethat producing intermediate summaries can save the work involved inre-running the query for previous time periods, so only the newer eventdata needs to be processed while generating an updated report. Thesereport acceleration techniques are described in more detail in U.S. Pat.No. 8,589,403, issued on Nov. 19, 2013, and U.S. Pat. No. 8,412,696,issued on Apr. 2, 2011.

Performing Searches Based on Qualitative Search Terms

A major challenge in processing a query containing a qualitative searchterm is to translate the qualitative search term into a mathematicalrepresentation that can be applied to numerical values in a data set.This can be accomplished by using a concept-mapping function as isillustrated in FIG. 7A. This concept-mapping function represents theconcept “ontime” by associating specific arrival times (indicated on thex axis) with compatibility index values between 0 and 1 (indicated onthe y axis). As illustrated in FIG. 7A, when an arrival time varies froma scheduled arrival time 710, the corresponding compatibility indexdecreases from a maximum value of 1 toward a value of zero. Thisconcept-mapping function approximates the way a user thinks about theconcept “ontime,” because the closer the arrival time is to thescheduled arrival time 710, the more likely a user will be to considerthe arrival time to be “ontime.” For example, a user is very likely toconsider an arrival time within one minute of a scheduled arrival to be“ontime,” whereas the user is considerably less likely to consider anarrival time 15 minutes past the scheduled arrival time to be ontime.Note that the shapes of concept-mapping functions can vary amongdifferent users and different contexts as is discussed in further detailbelow.

The disclosed embodiments combine a set of related concepts that applyto an attribute into an entity referred to as a “context” as isillustrated in FIG. 7B. The context illustrated in FIG. 7B includes fiveoverlapping concept-mappings related to the attribute “arrival time.”These concept-mappings represent the concepts “unusual,” “early,”“ontime,” “late,” and “unacceptable.” More specifically, from left toright, the first solid line segment 701, which drops from 1 to 0,represents the concept “unusual.” The next overlapping dashed linesegment 702, which rises from 0 to 1 and then falls to 0, represents theconcept “early.” The next overlapping solid line segment 703, whichrises from 0 to 1 and then falls back to 0, represents the concept“ontime.” The next, overlapping dashed line segment 704, which risesfrom 0 to 1 and then back falls to 0, represents the concept “late.”Finally, the last overlapping solid line segment 705, which rises from 0to 1, represents the concept “unacceptable.”

Note that adjoining concept-mapping functions within a context canoverlap because a user is likely to consider values between adjoiningconcepts to be related to both concepts. For example, a user mayconsider an arrival time of nine minutes past a scheduled arrival timeto be both somewhat ontime and somewhat late. In some embodiments, thesum of the compatibility indices at each point in these overlappingregions equals 1.

The above-described technique can be modified by applying linguistichedges to a concept-mapping function as is illustrated in FIG. 7C. Inthis example, applying the linguistic hedge “extremely” to thequalitative search term “ontime” causes the distribution of theresulting concept-mapping to be sharpened to produce the functionrepresented by the dashed line 706, in which only arrival times that arevery close to the scheduled arrival time 710 have a compatibility indexclose to 1. In contrast, applying the linguistic hedge “somewhat” to thequalitative search term “ontime” causes the distribution of theresulting concept-mapping to be more disbursed to produce the functionrepresented by the dashed line 707, in which arrival times that arefarther from the scheduled arrival time 710 can have a compatibilityindex close to 1. In general, hedges can be associated with any possiblelinguistic modifier for a qualitative search term. For, example hedgescan include terms such as “very,” “extremely,” “somewhat,” “not,” “closeto,” “above,” “below,” and “quite.” Moreover, multiple hedges can beapplied to a single search term. For example, the string of hedges“extremely close to below” can be applied to the qualitative search term“ontime.” Note that the possible types of hedges are, in theory,unlimited. This can be achieved by using “synonyms”, which map words tohedges. For example, a query can say “darn high,” where “darn” is asynonym for “very”.

These concept-mappings and associated hedges are related to “fuzzylogic,” which was developed to model cognitive systems. (See Cox, E. D.,“Solving Problems with Fuzzy Logic,” AI Expert, January 1992, Pages40-45.) Although fuzzy logic has been discussed extensively in theacademic literature, because of challenges relating to computationalrequirements, development of query languages, and management ofcontextual information, it has so far not been effectively applied inpractical query-processing systems that operate on large data sets.

The disclosed embodiments address these challenges by representingconcept-mappings and associated contexts using a number of datastructures that are illustrated in FIG. 8. As illustrated in FIG. 8,each concept-mapping is represented as a data structure, such asconcept-mappings 801-804. Moreover, one or more related concept-mappingscan be incorporated into each context. For example, concept-mappings801-804 are incorporated into context 810. Also, a set of relatedcontexts can be stored in the same container. For example, relatedcontexts 810-812 are stored in container 820.

Each of the concept-mappings 801-804 illustrated in FIG. 8 can berepresented using a data structure that specifies attributes of theconcept-mapping, including: (1) the name of the associated qualitativesearch term (e.g., “late”), (2) an identifier for the attribute that thequalitative search term applies to (e.g., arrival time), and (3) arepresentation of the concept-mapping function. In some embodiments, theconcept-mapping function is represented using an array of x and ycoordinates that describes the shape of the concept-mapping function.For example, the concept-mapping function for the qualitative searchterm “ontime” illustrated in FIG. 7A can be represented using 256equally spaced points along the x axis, wherein each point has acorresponding y value on the curve 703, which represents the concept“ontime.” These (x,y) coordinate pairs can be stored in an arraycontaining 256 entries for coordinate pairs. Each (x,y) coordinate pairin this array associates an attribute value x with a correspondingcompatibility index y. Representing a concept-mapping function in thisway enables the system to rapidly look up a compatibility index for agiven attribute value.

Instead of using an array, the concept-mapping function canalternatively be represented using a mathematical function that mapsattribute values to corresponding compatibility indices. However, afunctional representation of this type can require more mathematicaloperations to be performed at search time as compared with a simplearray lookup. However, an array of data values is likely to occupyconsiderably more space than a representation of a function.

The disclosed embodiments can also create a one-concept context for ascalar value that appears in a query. For example if the scalar value 65appears in a query, a context can be created with a singleconcept-mapping function that has non-zero compatibility indices overthe range 65±2.5%. This facilitates using a scalar value in combinationwith other qualitative search terms that are applied to other attributevalues in the same query. Note that the +/−2.5% range is a defaultrange, but is also configurable.

A number of related concept-mapping data structures can be combined intodata structures referred to as a “context,” wherein the context canspecify: (1) a context name, (2) a class identifier, (3) the type ofcontext, (4) the names of the concept-mappings contained in the context,(4) the shapes of concept-mapping functions in the context, and (6)other instructions that specify how to generate the context. Morespecifically, the context name (e.g., “arrival time”) identifies thespecific context. The class identifier specifies a class of events towhich the context applies (e.g., gender=female). Recall that this classinformation is specified in one or more fields in each event record. Thetype of context (e.g., “domain” or “average centered” or “mediancentered”) specifies how the bounds of the context are determined, as isdiscussed in more detail below. The names of the concepts (e.g.,“early,” “ontime,” and “late”) identify the concept-mappings containedin the context. The shapes of the concept-mapping functions (e.g.,“linear,” “trapezoidal,” “triangular”) can be defined by a domainexpert; alternatively, default shapes may be used. Finally, the otherinstructions can specify a search that is used to obtain a set of eventsthat are used to generate the context.

Finally, a collection of related contexts can be incorporated into acontainer 820, which comprises a file that contains a collection ofcontexts that can possibly be related (but are not necessarily related).For example, a container can contain user-specific contexts associatedwith a specific user, or organization-specific contexts associated witha specific organization. This allows the system to maintain customizedsets of the contexts for different users or organizations. Also, thename of a specific container can be indicated in a query, which enablesthe query to use a customized set of contexts for qualitative searchterms.

In some cases, a container can include different class-specific contextsfor a given attribute. For example, an exemplary container can includeclass-specific contexts for the attribute “height,” including a contextfor “male height,” which is specific to the class gender=male; a contextfor “female height,” which is specific to the class gender=female; and adefault context for “height,” which is not gender-specific. This enablesa query to be applied using gender-specific contexts based on a genderindicated in a corresponding gender field in each event. In anotherexample, a container can include class-specific contexts for differenttypes of operating systems for computers, including contexts for“operating system=Windows,” “operating system=Linux,” “operatingsystem=OS X” and a default context. This enables a query to be appliedusing operating-system-specific contexts based on an operating systemindicated in a corresponding operating-system field in each event.

Context Variations

In describing the ways in which a context can constructed, one candistinguish between a necessary process and an example of a possibletechnique. There are three areas of technique that can be alluded to butnot made a specific case of necessity: (1) the ways in which theunderlying qualitative semantics (fuzzy sets) are generated, (2) theshape of the fuzzy set (e.g., concept-mapping function), and (3) theorder of the fuzzy sets.

We first describe the ways in which fuzzy sets can be created. When wecreate an adaptive context, the number, shapes, and order of theunderlying qualitative semantics (fuzzy sets) are determined by somefunction (or set of functions) that characterizes the data. Thesefunctions can be or can be derived from analytical, probabilistic,evolutionary, or subjective processes (or any other process that, in thefuture might be generated by some other machine intelligence).Analytical means that the function is an algorithm, a heuristic, or ablending of the two. Probabilistic means that the characteristic isderived from the probabilities (and cumulative probability distributionfunctions, perhaps) associated with, as an example, incoming sets ofevents (such as Bayes Theorem or Markov Models). Evolutionary means thatthat the characteristic is derived from an exploration of how acollected set of parameters can be simultaneously evaluated to fund thebest (optimal or near optimal) set of characteristics (in this case weuse evolutionary strategies such as genetic algorithms, Monte Carolsimulation, and so forth). Subjective means that the analyst simplypicks defining characteristic properties that he/she/it feels is a goodapproximation of the data (the “it” hear means that the subject matterexpert (analyst) might also be an intelligent or non-intelligent machinesystem, such as co-operating cellular automata networks). Any of theseapproaches might also take into account the time varying or datadimensional periodicity of the data.

Just to give a brief example, techniques can be based on any statisticalor mathematical property (or set of properties), not necessarily limitedto the average or even the mode. As just a few quick examples, when wesay “average” we mean the entire computational class of averages:arithmetic, weighted, geometric, and so forth. But an underlyingcollection of qualitative semantics can be generated in many ways fromany functional, algorithmic, or heuristic methodology. In a multi-modaldistribution, as an example, each mode, perhaps weighted by itsnormalized, relative frequency to all the other modes, represents ananchor fuzzy set. The other fuzzy sets (semantics) are then generated(perhaps to the left and right) from theses anchor sets, hence creatinga set of semantics representing the “centers of data intensity” in thecontext. But the parameters that characterize the data (and from whichthe context is generated) can be drawn from any set of statisticalproperties as well as deterministic as well as stochastic processes. Asan example we might pick a random number drawn from the descriptiveproperties of the data, or we might use a Monte Carlo or geneticalgorithm to explore the best possible partitioning of the data. Infact, in keeping with an information theory approach to handlingentropy, we might inject a context with some kind of random backgroundnoise (uniform, Gaussian, etc) and then construct semantics on top ofthis noise layer.

We next describe the shapes. Cognitive based qualitative semantics are,in most cases, naturally defined by bell-shapes and sigmoid (S) shapes.These bell-shaped fuzzy sets are not necessarily derived from thestatistical properties of the data. But the shape of a fuzzy set can bearbitrary (defined by a set of points, as an example) as well as atrapezoid, a triangular, a singleton, and so forth. Fuzzy sets of manydifferent shapes can also be generated from a scalar or a set ofscalars. Fuzzy sets are not always symmetrical. They can have irregularshapes (like a skew and kurtosis in a bell curve or a difference inangle of the sides in trapezoids, and so forth).

We finally describe the order. The order of a fuzzy set is determined bythe nature of the values on the domain and the membership axes.Currently we are using first-order fuzzy sets—the values on each axisare ordinary numbers. In a second order fuzzy set, the values of eachaxis is a fuzzy set. This creates a fuzzy set whose membership functionis a band of different “densities.” Second order fuzzy sets will be veryimportant in cognitive models. There are higher order fuzzy sets and wereserve them as part of the patent.

There are also other factors that are used to characterize a collectionof qualitative semantics in a context: the amount of overlap betweenneighboring fuzzy sets, the alpha cut for determining the strength ofthe fuzzy set, the method of decomposition and aggregation indetermining the compatibility index, and so forth.

Commands to Facilitate Conceptual Searches

The above-described event-based query-processing system can be augmentedto facilitate searches based on qualitative search criteria byincorporating a number of additional commands into the search language(e.g., SPL). For example, FIG. 9A illustrates a number ofconceptual-search commands that can be incorporated into the searchlanguage in accordance with the disclosed embodiments. The mostfundamental conceptual-search command is the “xswhere” command thatoperates like a normal “where” command in SPL, except that itfacilitates looking for events based on one or more qualitative searchterms.

A number of these conceptual-search commands can be used to performoperations on individual concepts or contexts, including data-drivencommands to clone, create, delete or display concepts and contexts.There also exists an “xsCreateUDContext” command to facilitate creatinga user-defined context. The command “xsGetWhereCIX” computes one or morecompatibility indices for an event and then forwards the computedcompatibility indices along with the event to facilitate downstreamprocessing of the event. The conceptual-search commands also includecommands to list concepts and contexts as well as the command“xsListUOM,” which lists units of measure (e.g., “centimeters” or“kilograms”) for specific attributes associated with contexts. Therealso exist commands to merge, rename and update contexts.

The “xsDiscoverTrend” command facilitates discovering trends in specificattribute values associated with contexts. It operates by running aregression over attribute values in the events to determine a functionthat represents the attribute values. The “xsDisplayWhere” commanddisplays the resulting concept-mapping function for an “xsWhere” query,possibly including hedges. The “xsFindBestConcept” command returns aconcept from a context that best matches a specific attribute value. The“xsFindMembership” command computes a compatibility index for eachconcept-mapping in a context, and then adds a field for each of thecomputed indices to the event and forwards the computed compatibilityindices along with the event to facilitate downstream processing. Thecommand “xsUpdateUDContext” updates a user-defined context. (Note thatassociating zero events with a context prevents the system from updatingthe context.) There also exist macros, which are not preceded by theprefix “xs,” comprising sequences of commands that perform operationsincluding: finding the best concept, creating a context and discoveringa trend.

There also exist statistical-reasoning commands that are illustrated inFIG. 9B. These statistical-reasoning commands perform standardcorrelations and regressions. Statistical-reasoning commands that areprefixed with “xsPerform” generate an algorithm given a set of events.Statistical-reasoning commands that are prefixed with “xsApply” applythe algorithm to future events. Statistical-reasoning commands that areprefixed with “xsAggregate” can be used to combine results from the“xsPerform” commands in a weighted fashion if the underlying functionsare additive. For example, the system can perform a linear regressionevery hour, and these hourly linear regressions can be aggregated overan entire day to produce a linear regression for the day. The“xsPredict” command facilitates predicting future attribute values.Finally, the “xsGetDistance” command implements the Haversinedistance-computation technique to facilitate processing queries thatinvolve distances.

Query Processing Operations

FIG. 10 presents a flow chart illustrating how a query containing aqualitative search term can be processed in accordance with thedisclosed embodiments. First, the system receives a query to beprocessed, wherein the query specifies a qualitative search term to beapplied to an attribute (step 1002). Note that commands associated withqualitative search terms can be chained together in SPL by using pipesymbols “I”.

Then, for each event in the set of events, the system performs thefollowing operations. First, the system extracts an attribute valueassociated with the qualitative search term from the event (step 1006).Next, the system looks up a context-mapping for the qualitative searchterm. If the query specifies that the qualitative search term is to beapplied in a class-specific manner, the system determines what class theevent belongs to by examining one or more fields in the event, and thenlooks up a class-specific concept-mapping for the qualitative searchterm based on the determined class (step 1008). In this way a singlequery can apply different class-specific contexts to each eventdepending upon which class is specified by fields in the event.

Next, the system uses the concept-mapping for the qualitative searchterm to determine a compatibility index between the attribute value andthe qualitative search term (step 1010). Finally, the system uses thedetermined compatibility index as a factor in determining whether toinclude the event in a set of query results (step 1012).

FIG. 11 presents a flow chart illustrating how linguistic hedges can beprocessed in accordance with the disclosed embodiments. First, thesystem obtains linguistic hedges for qualitative search terms from aquery (step 1102). This can take place when the system is parsing thequery. Next, the system resolves synonyms for linguistic hedges (step1104). This can involve performing a lookup in a table containingsynonyms and associated hedges. Finally, the system modifies theconcept-mapping for each qualitative search term in accordance with theassociated linguistic hedges (step 1106). Techniques for mathematicallymodifying such concept-mappings based on linguistic hedges are wellknown in the art. For example, see De Cock, M. and Kerre, E. E., “AContext-Based Approach to Linguistic Hedges,” Int. J. Appl. Math. Comp.Sci., 2002, Vol. 12, No. 3, 371-382.

FIG. 12 presents a flow chart illustrating how compatibility indicesassociated with different qualitative search terms can be combined witheach other during query processing in accordance with the disclosedembodiments. If the query includes multiple qualitative search terms,the system first determines compatibility indices for the multiplequalitative search terms for each event (step 1202). Next, the systemcombines the determined compatibility indices into an aggregatecompatibility index for each event (step 1204). For example, thecompatibility indices can be combined by computing an average of thecompatibility indices, or alternatively by computing a weighted averageof the compatibility indices based on an ordering of associatedqualitative search terms in the query. Note that qualitative searchterms that appear earlier in the query are likely to be more importantthan qualitative search terms that appear later in the query. Hence, thesystem can weight the earlier qualitative search terms more highly thanthe later qualitative search terms while computing the average. Finally,the system determines whether to include each event in the set of queryresults based on whether the aggregate compatibility index for the eventmeets or exceeds a threshold (step 1206). For example, this thresholdcan be the 0.2 value, which is associated with an “alpha cut.”

Context-Updating Operations

FIG. 13 presents a flow chart illustrating how a context can be updatedin accordance with the disclosed embodiments. First, the system obtainsinstructions specifying how to update a context associated with anattribute (step 1302). In some cases, these instructions can be obtainedfrom the context itself, which makes the context “self aware,” meaningthat the context is aware of how it was created. Next, the systemextracts associated attribute values from a set of events (step 1304).

The system then uses the extracted attribute values to determineboundaries for the context (step 1306). Note that each concept-mappingwithin a context has the same boundaries (start point and end point).The boundaries for a “domain context” can be determined by looking atthe range of attribute values contained in the set of events, whereinthe minimum attribute value and the maximum attribute value in the rangecan be used to define the boundaries of the domain. The boundaries foran “average-centered context” can be determined by computing an averageattribute value and a standard deviation based on the extractedattribute values, and then radiating outward from the average valuebased on the standard deviations to determine boundaries for the domain.The boundaries for a “mean-centered context” can be determinedsimilarly, except that the mean value is used in place of the averagevalue. The system can also employ a user-specified center value todetermine the boundaries of a context.

The width of the context can be computed in a manner that takes intoaccount the number of overlapping concepts contained in the event. Forexample, if the context (e.g., arrival time) includes five overlappingconcepts (e.g., unusual, early, ontime, late, unacceptable), whereineach concept-mapping function is two standard deviations wide, andadjacent concept-mapping functions overlap by one standard deviation,then the entire context will be six standard deviations wide. Hence, theboundaries of a corresponding average-centered context can be computedusing the following equations:max=avg(x)+sdev(x)*numConcepts; andmin=avg(x)−sdev(x)*numConcepts.

Note that the system also supports median-centered contexts, wherein amedian-centered context uses median(x) instead of avg(x) as its center.

In some embodiments, the width of a context is determined by computing aweighted average of the width of the previous context and a widthcomputed based on the extracted attribute values.

Finally, within the determined boundaries, the system can usepre-specified distributions to update each concept-mapping function inthe context (step 1308). Note that the shapes of these pre-specifieddistributions for the concept-mapping functions can be defined by adomain expert. Also, note that a context can be updated manually orthrough a periodically scheduled update.

The preceding description was presented to enable any person skilled inthe art to make and use the disclosed embodiments, and is provided inthe context of a particular application and its requirements. Variousmodifications to the disclosed embodiments will be readily apparent tothose skilled in the art, and the general principles defined herein maybe applied to other embodiments and applications without departing fromthe spirit and scope of the disclosed embodiments. Thus, the disclosedembodiments are not limited to the embodiments shown, but are to beaccorded the widest scope consistent with the principles and featuresdisclosed herein. Accordingly, many modifications and variations will beapparent to practitioners skilled in the art. Additionally, the abovedisclosure is not intended to limit the present description. The scopeof the present description is defined by the appended claims.

The data structures and code described in this detailed description aretypically stored on a computer-readable storage medium, which may be anydevice or medium that can store code and/or data for use by a system.The computer-readable storage medium includes, but is not limited to,volatile memory, non-volatile memory, magnetic and optical storagedevices such as disk drives, magnetic tape, CDs (compact discs), DVDs(digital versatile discs or digital video discs), or other media capableof storing code and/or data now known or later developed.

The methods and processes described in the detailed description sectioncan be embodied as code and/or data, which can be stored on anon-transitory computer-readable storage medium as described above. Whena system reads and executes the code and/or data stored on thenon-transitory computer-readable storage medium, the system performs themethods and processes embodied as data structures and code and storedwithin the non-transitory computer-readable storage medium.

Furthermore, the methods and processes described below can be includedin hardware modules. For example, the hardware modules can include, butare not limited to, application-specific integrated circuit (ASIC)chips, field-programmable gate arrays (FPGAs), and otherprogrammable-logic devices now known or later developed. When thehardware modules are activated, the hardware modules perform the methodsand processes included within the hardware modules.

What is claimed is:
 1. A method for executing a query, comprising:receiving the query at a computer system, wherein the query applies aone or more qualitative search terms to an attribute of data items in aset of data items; and while executing the query on the computer system,processing each data item in the set of data items by, extracting anattribute value from the data item, using a concept-mapping to determinea compatibility index for the attribute value, wherein theconcept-mapping associates each attribute value with a numericalcompatibility index that indicates a compatibility between the attributevalue and the one or more qualitative search terms, wherein the conceptmapping is represented using an array containing X and Y coordinatepairs describing a shape of a concept-mapping function, wherein usingthe concept-mapping to determine the compatibility index includes usingthe attribute value to perform a lookup in the array to retrieve thecompatibility index, and using the compatibility index as a factor indetermining whether to include the data item in a set of query results;and wherein when the query includes multiple qualitative search terms,executing the query comprises, determining compatibility indices for themultiple qualitative search terms for each data item, combining thedetermined compatibility indices into an aggregate compatibility indexfor each data item, wherein combining the determined compatibilityindices involves one of computing an average for the multiplecompatibility indices, and computing a weighted average for the multiplecompatibility indices based on an ordering of associated qualitativesearch terms in the query, and determining whether to include each dataitem in the set of query results based on whether the aggregatecompatibility index for the data item meets or exceeds a threshold. 2.The method of claim 1, wherein the concept-mapping is part of a contextcomprising a set of concept-mappings associated with a set ofqualitative search terms for the attribute; and wherein using theconcept-mapping to determine the compatibility index involves using theset of concept-mappings to determine a set of compatibility indices forthe set of qualitative search terms associated with the attribute. 3.The method of claim 1, wherein the concept-mapping is a class-specificconcept-mapping; and wherein prior to using the concept-mapping, themethod further comprises selecting the concept-mapping based on classinformation obtained from one or more fields in the set of data items.4. The method of claim 1, wherein the concept-mapping is stored in acontainer file that contains a related set of concept-mappings.
 5. Themethod of claim 1, wherein the concept-mapping is stored as an array ofdata values representing a function that maps attribute values tocorresponding compatibility indices.
 6. The method of claim 1, whereinthe concept-mapping is part of a context comprising a set ofconcept-mappings associated with a set of qualitative search terms forthe attribute; and wherein the method further comprises updating thecontext at periodic or aperiodic intervals based on instructionscontained in the context.
 7. The method of claim 1, wherein when thequery includes a linguistic hedge that modifies the one or morequalitative search terms, the method further comprises modifying theconcept-mapping in accordance with the linguistic hedge prior to usingthe concept-mapping.
 8. The method of claim 1, wherein when the queryincludes a linguistic hedge that modifies the one or more qualitativesearch terms, the method further comprises modifying the concept-mappingin accordance with the linguistic hedge; and wherein the processing ofthe query includes resolving synonyms for the linguistic hedge prior tousing the linguistic hedge to modify the concept-mapping.
 9. The methodof claim 1, wherein when the query includes a scalar value, the methodfurther comprises creating a concept-mapping for the scalar value thatassociates non-zero compatibility indices with attribute values thatfall within a specified range of the scalar value.
 10. A non-transitorycomputer-readable storage medium storing instructions that when executedby a computer cause the computer to perform a method for executing aquery, the method comprising: receiving the query, wherein the queryapplies one or more qualitative search terms to an attribute of dataitems in a set of data items; and while executing the query, processingeach data item in the set of data items by, extracting an attributevalue from the data item, using a concept-mapping to determine acompatibility index for the attribute value, wherein the concept-mappingassociates each attribute value with a numerical compatibility indexthat indicates a compatibility between the attribute value and the oneor more qualitative search terms, wherein the concept mapping isrepresented using an array containing X and Y coordinate pairsdescribing a shape of a concept-mapping function, wherein using theconcept-mapping to determine the compatibility index includes using theattribute value to perform a lookup in the array to retrieve thecompatibility index, and using the compatibility index as a factor indetermining whether to include the data item in a set of query results;and wherein when the query includes multiple qualitative search terms,executing the query comprises, determining compatibility indices for themultiple qualitative search terms for each data item, combining thedetermined compatibility indices into an aggregate compatibility indexfor each data item, wherein combining the determined compatibilityindices involves one of computing an average for the multiplecompatibility indices, and computing a weighted average for the multiplecompatibility indices based on an ordering of associated qualitativesearch terms in the query, and determining whether to include each dataitem in the set of query results based on whether the aggregatecompatibility index for the data item meets or exceeds a threshold. 11.The non-transitory computer-readable storage medium of claim 10, whereinthe concept-mapping is part of a context comprising a set ofconcept-mappings associated with a set of qualitative search terms forthe attribute; and wherein using the concept-mapping to determine thecompatibility index involves using the set of concept-mappings todetermine a set of compatibility indices for the set of qualitativesearch terms associated with the attribute.
 12. The non-transitorycomputer-readable storage medium of claim 10, wherein theconcept-mapping is a class-specific concept-mapping; and wherein priorto using the concept-mapping, the method further comprises selecting theconcept-mapping based on class information obtained from one or morefields in the set of data items.
 13. The non-transitorycomputer-readable storage medium of claim 10, wherein theconcept-mapping is stored in a container file that contains a relatedset of concept-mappings.
 14. The non-transitory computer-readablestorage medium of claim 10, wherein the concept-mapping is stored as anarray of data values representing a function that maps attribute valuesto corresponding compatibility indices.
 15. The non-transitorycomputer-readable storage medium of claim 10, wherein theconcept-mapping is part of a context comprising a set ofconcept-mappings associated with a set of qualitative search terms forthe attribute; and wherein the method further comprises updating thecontext at periodic or aperiodic intervals based on instructionscontained in the context.
 16. The non-transitory computer-readablestorage medium of claim 10, wherein when the query includes a linguistichedge that modifies the one or more qualitative search terms, the methodfurther comprises modifying the concept-mapping in accordance with thelinguistic hedge prior to using the concept-mapping.
 17. Thenon-transitory computer-readable storage medium of claim 10, whereinwhen the query includes a linguistic hedge that modifies the one or morequalitative search terms, the method further comprises modifying theconcept-mapping in accordance with the linguistic hedge; and wherein theprocessing of the query includes resolving synonyms for the linguistichedge prior to using the linguistic hedge to modify the concept-mapping.18. The non-transitory computer-readable storage medium of claim 10,wherein when the query includes a scalar value, the method furthercomprises creating a concept-mapping for the scalar value thatassociates non-zero compatibility indices with attribute values thatfall within a specified range of the scalar value.
 19. A system,comprising: at least one processor and at least one associated memory; aquery processor that executes on the at least one processor and receivesthe query, wherein the query applies one or more qualitative searchterms to an attribute of data items in a set of data items; and whileexecuting the query, the query processor processes each data item in theset of data items by, extracting an attribute value from the data item,using a concept-mapping to determine a compatibility index for theattribute value, wherein the concept-mapping associates each attributevalue with a numerical compatibility index that indicates acompatibility between the attribute value and the one or morequalitative search terms, wherein the concept mapping is representedusing an array containing X and Y coordinate pairs describing a shape ofa concept-mapping function, wherein using the concept-mapping todetermine the compatibility index includes using the attribute value toperform a lookup in the array to retrieve the compatibility index, andusing the compatibility index as a factor in determining whether toinclude the data item in a set of query results; and wherein when thequery includes multiple qualitative search terms, the query processor,determines compatibility indices for the multiple qualitative searchterms for each data item, combines the determined compatibility indicesinto an aggregate compatibility index for each data item, whereincombining the determined compatibility indices involves one of computingan average for the multiple compatibility indices, and computing aweighted average for the multiple compatibility indices based on anordering of associated qualitative search terms in the query, anddetermines whether to include each data item in the set of query resultsbased on whether the aggregate compatibility index for the data itemmeets or exceeds a threshold.
 20. The system of claim 19, wherein theconcept-mapping is part of a context comprising a set ofconcept-mappings associated with a set of qualitative search terms forthe attribute; and wherein while using the concept-mapping to determinethe compatibility index, the query processor is configured to use theset of concept-mappings to determine a set of compatibility indices forthe set of qualitative search terms associated with the attribute. 21.The system of claim 19, wherein the concept-mapping is a class-specificconcept-mapping; and wherein prior to using the concept-mapping, thequery processor is configured to select the concept-mapping based onclass information obtained from one or more fields in the set of dataitems.
 22. The system of claim 19, wherein the concept-mapping is storedin a container file that contains a related set of concept-mappings. 23.The system of claim 19, wherein the concept-mapping is stored as anarray of data values representing a function that maps attribute valuesto corresponding compatibility indices.
 24. The system of claim 19,wherein the concept-mapping is part of a context comprising a set ofconcept-mappings associated with a set of qualitative search terms forthe attribute; and wherein the query processor is further configured toupdate the context at periodic or aperiodic intervals based oninstructions contained in the context.